Byzantine-Resilient Decentralized Stochastic Gradient Descent

Decentralized learning has gained great popularity to improve efficiency and preserve data privacy. Each computing node makes equal contribution collaboratively learn a Deep Learning model. The elimination of centralized Parameter Servers (PS) can effectively address many issues such as privacy, performance bottleneck single-point-failure. However, how achieve Byzantine Fault Tolerance in decentralized systems is rarely explored, although this problem been extensively studied systems. In paper, we present an in-depth study towards the resilience with two contributions. First, from adversarial perspective, theoretically illustrate that attacks are more dangerous feasible systems: even one malicious participant arbitrarily alter models other participants by sending carefully crafted updates its neighbors. Second, defense propose Ubar, novel algorithm enhance Tolerance. Specifically, Ubar provides U niform xmlns:xlink="http://www.w3.org/1999/xlink">B yzantine-resilient xmlns:xlink="http://www.w3.org/1999/xlink">A ggregation xmlns:xlink="http://www.w3.org/1999/xlink">R ule for benign nodes select useful parameter filter out ones each training iteration. It guarantees system train correct model under very strong arbitrary number faulty nodes. We conduct extensive experiments on standard image classification tasks results indicate defeat both simple sophisticated higher than existing solutions.